[Benchmark] Add asyncio-based multi_turn benchmark v2 to fix deadlock - #53075

Purpose

Fixes #42226.

The original benchmark_serving_multi_turn.py deadlocks when mp.Queue.join_thread() waits for a feeder thread that cannot flush a full OS pipe (~1 MB on Linux). mp.Queue.empty() is documented as unreliable, so the queue drain loop exits prematurely, leaving data in the pipe that join_thread() then waits for indefinitely. The deadlock is deterministic when conversation data is large enough (30-40 turns, ~30-80 KB per conversation pickled) and intermittent with smaller conversations.

This PR adds benchmark_serving_multi_turn_v2.py (new file, original unchanged) that replaces the multiprocessing architecture with a single-process asyncio event loop, eliminating all mp.Queue usage.

Why not patch the existing file?

PR #42327 (CLOSED) attempted a conservative patch (get_nowait + cancel_join_thread). This only fixes the join_thread() deadlock but leaves other deadlock paths:

1. task_queue.get() — blocking call with no timeout; clients stuck here cannot check stop_event
2. conv_queue.get() — main process blocks indefinitely if a client crashes without sending TERM_SIGNAL
3. result_queue pipe buffer backpressure — clients block on put() when the pipe is full and the main process isn't draining fast enough

The asyncio rewrite eliminates all of these by design:

• asyncio.wait_for(task_queue.get(), timeout=1.0) + stop_event check
• asyncio.wait(client_tasks, timeout=5.0) monitoring loop detects task.exception()
• Shared in-process list/dict (no pipe, no backpressure)
• --benchmark-timeout-sec global safety net

Additional features

• --generate-only: generate dataset to JSON without running the benchmark (generate once, reuse for multiple runs)
• Shared aiohttp.ClientSession with TCPConnector (pattern from vllm/benchmarks/serve.py)
• Per-client RNG instances (avoids shared global state in asyncio)

Test Plan

Tested on 2× Ascend 910B3 NPU with Qwen3-0.6B (--data-parallel-size 2), VLLM_SERVER_DEV_MODE=1, --enable-prefix-caching. Cache cleared between tests via POST /reset_prefix_cache.

Linter commands:

python -m ruff check benchmarks/multi_turn/benchmark_serving_multi_turn_v2.py
python -m ruff format --check benchmarks/multi_turn/benchmark_serving_multi_turn_v2.py

Both pass.

Test phases:

1. Low concurrency performance comparison (v1 vs v2) — 30 conversations, 8-10 turns, --no-early-stop
2. Deadlock detection — 2000 conversations, 30-40 turns, --max-num-requests 200
3. v2 upper limit — up to 2000 concurrent clients + sustained load

Test Result

Phase 1: Low concurrency (small conversations — both pass)

At 4 clients, performance is within normal variance. v1's higher mean TTFT at 2 clients is from multiprocessing startup overhead (p90 is comparable: 86.6 vs 78.2ms).

Phase 2: Deadlock detection (large conversations)

v1 deadlocks at all concurrency levels (2-100c) with 30-40 turn conversations. v2 completes at all levels.

Phase 3: v2 upper limit

v2 did not crash at 2000 concurrent clients. Peak throughput ~19 RPS. Server saturation is the bottleneck, not the client.

Sustained load (100c x 4000 requests, 50c x full 2000 conversations): ran 9+ minutes, graceful timeout exit, no memory leaks.

This PR was prepared with AI assistance (GLM5.2 + ZCode). All code was reviewed and all tests were run by the human submitter.

• The purpose of the PR (link to issue [Bug]: benchmark_serving_multi_turn.py deadlocks after clients exit when --max-num-requests is used #42226)
• The test plan (test commands)
• The test results (before/after comparison and e2e results)